This post is not strictly about Perl, but it is technical and about automated testing, which is a hot topic nowadays in the Perl world, so I decided to post it here.
I have written an ANSI C library called "Freecell Solver", which solves several variants of card solitaire. Now, the problem is that it doesn't have any automated tests yet, and I'd like to add some before I hack on it further.
Now, having attended Ori Peleg's presentation "Thoughts on Testing" on three different occassions, I became convinced that testing can only be done with dynamic languages, and writing complex test logic in C or Java should be avoided. Now, I may still be able to use perl for generating some simple C code for unit tests, that will then be compiled, or for system tests of the solver. However, it will probably not be very suitable for more complex unit or integration tests, because of its complex C extension API. So I'm looking for a better alternative.
One thing I've decided is that all my tests will produce TAP output, so I can use a good TAP analyser like Test::Run or TAPx::Parser, to process all the test scripts or a subset of them. If necessary, I will write a rudimentary TAP-outputting test framework in the language I choose.
Let me enumerate the options I see and what I think of them:
Perl with XS - like I said - too complex and will require a lot of coding.
Perl with Inline::C. This could hide away some of the complexity but I believe some complexity will still remain, as it still requires writing some code using XS.
I could use SWIG to write my bindings. My former experience with SWIG, however, was not entirely positive as it doesn't handle pointers meaning arrays and other Cisms well, and often necessitates writing some glue code. By using SWIG, I could still write my unit tests in Perl.
Python - I'm not entirely fond of it and think I'd like Ruby better.
Ruby - I was told it has an even better extensibility API than Python, and it's also more similar to Perl. I do not intend any normal end-user to run the test suite so I can rely on Ruby being around.
Tcl - should have a good extensibility API, but from what I know strings are null-terminated, and I dislike it quite a bit, so if at all - Ruby would be preferable. I mentioned it here for completeness' sake.
Lua - small and with a good extensibility API. I don't think it's as rich as Ruby or Perl, but it probably won't matter too much for testing.
The Io Language - similar to Lua in size. It's very interesting as a language, but I had some problems deploying it properly on my system, and I dislike its syntax for its "[object][space]method" convention and other mis-conventions like that.
Smalltalk - the Squeak Virtual machine has a mind of its own, but there's GNU Smalltalk which works better with C. However, I dislike the Smalltalk syntax, and GNU Smalltalk is still a bit buggy, and I would probably prefer Ruby or Lua.
Haskell and O'Caml are strongly typed and which will be an obstacle for testing.
Scheme - I could use Guile or TinyScheme. However, Scheme tends to be overly verbose and to lack a lot of nice syntactic sugar. The same can be said of Common Lisp, but there there's also a lack of a decent ubiquitous implementation.
Maybe I missed a few players, but I think I covered the most prominent ones. Like I said, all the tests will emit TAP so their results can be analysed and summarised with the Perl TAP tools. At the moment, I'm leaning towards using either Ruby or Lua, but would love to hear what you think about all this.
Note that I don't need Unicode or Localisation or an interface to any third-party library, as my library relies exclusively on the ANSI C library. But I do need to handle binary data (with '\0' and all) properly.
Re:No SX with P5NCI and Python's ctypes package
Shlomi Fish on 2007-02-05T20:19:56
ctypes looks very promising. Thanks for the recommendation. I'll also take a look at P5NCI.
Unfortunately, I haven't been able to attend "Thoughts on Testing." Is there any way you could summarize or provide a link to explain why you prefer to use a dynamic language to write tests for a C library? What language features are you looking for?
If you do in fact decide to write the tests in C, there's LibTap (tutorial).
Haskell and O’Caml are strongly typed and which will be an obstacle for testing.
Uh.
